An Integration Principle for Multimodal Sensor Data Based on Temporal Coherence of Self-Organized Patterns

The world around us offers continuously huge amounts of information, from which living organisms can elicit the knowledge and understanding they need for survival or well-being. A fundamental cognitive feature, that makes this possible is the ability of a brain to integrate the inputs it receives from different sensory modalities into a coherent description of its surrounding environment. By analogy, artificial autonomous systems are designed to record continuously large amounts of data with various sensors. A major design problem by the last is the lack of reference of how the information from the different sensor streams can be integrated into a consistent description. This paper focuses on the development of a sinergistic integration principle, supported by the synchronization of the multimodal information streams on temporal coherence principle. The processing of the individual information streams is done by a self organizing neural algorithm, known as Neural gas algorithm. The integration itself uses a supervised learning method to allow the various information streams to interchange their knowledge as emerged experts. Two complementary data streams, recorded by exploration of autonomous robot of unprepared environments are used to simultaneously illustrate and motivate in a concrete sense the developed integration approach.